1 


13  th  ICCRTS 

“C2  for  Complex  Endeavors” 

Title:  Effects  of  Visual  Communication  Tool  and  Separable  Status  Display  on  Team 
Performance  and  Subjective  Workload  in  Air  Battle  Management 

Topic  9:  Collaborative  Technologies  for  Network-Centric  Operations 
Topic  7:  Network-Centric  Experimentation  and  Analysis 
Topic  3:  Modeling  and  Simulation 


Point  of  Contact: 


Name  of  Organization: 
Address  of  Organization: 


Daniel  Schwartz 
937-657-6955 
Schwartz.22@wright.edu 
Daniel.schwartz@wpafb.af.mil 

Benjamin  A.  Knott 
937-785-8803 

Benjamin.Knott.wpafb.af.mil 

Scott  M.  Galster 

937-785-8737 

Scott.galster@wpafb.af.mil 

Air  Force  Research  Laboratory/Collaborative  Interfaces  Branch 
2255  H  Street 

Wright-Patterson  AFB,  OH  45433 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

JUN  2008 


2.  REPORT  TYPE 


4.  TITLE  AND  SUBTITLE 

Effects  of  Visual  Communication  Tool  and  Separable  Status  Display  on 
Team  Performance  and  Subjective  Workload  in  Air  Battle  Management 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Research  Laboratory, Collaborative  Interfaces  Branch, 2255  H 
Street,  W  right-Patterson  AFB , OH, 45433 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


3.  DATES  COVERED 

00-00-2008  to  00-00-2008 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

13th  International  Command  and  Control  Research  and  Technology  Symposia  (ICCRTS  2008),  17-19  Jun 
2008,  Seattle,  WA 

14.  ABSTRACT 

Tactical  Air  Battle  Managers,  such  as  AWACS  Weapons  Directors  (WDs),  perform  as  a  team  to  effect 
command  and  control  (C2)  of  assigned  forces  by  planning,  organizing,  and  directing  operations. 
Specifically,  AWACS  WDs  must  coordinate  offensive  counter-air,  defensive  counter-air,  and  air  refueling 
operations.  AWACS  WD  teams  accomplish  their  C2  function  through  networked  collaboration  that  is 
typically  supported  by  monitoring  multiple  radio  communications  channels  under  conditions  of  moderate 
to  high  ambient  cabin  noise  while  performing  several  visual  and  manual  tasks.  The  purpose  of  this  study  is 
to  compare  team  performance  and  subjective  workload  on  a  simulated  AWACS  scenario,  for  two 
conditions  of  communication  (Voice-only,  and  Voice  augmented  with  a  Visual  Communication  Tool),  and 
using  two  supplementary  display  conditions  (Separable  Status  Display  and  No-Separable  Status  Display). 
Team  performance  measures  on  the  AWACS  scenario  include  1)  the  percentage  of  enemy  targets  that  were 
allowed  to  penetrate  friendly  airspace,  2)  the  percentage  of  high  value  assets  destroyed  (i.e.,  the  air  base, 
infantry  units,  and  tanker  aircraft),  3)  the  percentage  of  lighter  assets  that  were  lost  due  to  fuel  depletion 
or  enemy  attack,  4)  the  average  time  of  enemy  target  prosecution. 

15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF 

ABSTRACT 

OF  PAGES 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

18 

2 


Abstract 

Tactical  Air  Battle  Managers,  such  as  AWACS  Weapons  Directors  (WDs),  perfonn  as  a  team  to 
effect  command  and  control  (C2)  of  assigned  forces  by  planning,  organizing,  and  directing 
operations.  Specifically,  AWACS  WDs  must  coordinate  offensive  counter-air,  defensive 
counter-air,  and  air  refueling  operations.  AWACS  WD  teams  accomplish  their  C2  function 
through  networked  collaboration  that  is  typically  supported  by  monitoring  multiple  radio 
communications  channels  under  conditions  of  moderate  to  high  ambient  cabin  noise  while 
performing  several  visual  and  manual  tasks.  The  purpose  of  this  study  is  to  compare  team 
performance  and  subjective  workload  on  a  simulated  AWACS  scenario,  for  two  conditions  of 
communication  (Voice-only,  and  Voice  augmented  with  a  Visual  Communication  Tool),  and 
using  two  supplementary  display  conditions  (Separable  Status  Display  and  No-Separable  Status 
Display).  Team  performance  measures  on  the  AWACS  scenario  include  1)  the  percentage  of 
enemy  targets  that  were  allowed  to  penetrate  friendly  airspace,  2)  the  percentage  of  high  value 
assets  destroyed  (i.e.,  the  air  base,  infantry  units,  and  tanker  aircraft),  3)  the  percentage  of  fighter 
assets  that  were  lost  due  to  fuel  depletion  or  enemy  attack,  4)  the  average  time  of  enemy  target 
prosecution. 
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Introduction 

Air  Battle  Managers  (ABMs)  effect  command  and  control  (C2)  of  assigned  forces  by 
planning,  organizing,  and  directing  operations.  Specifically,  ABMs  provide  friendly  forces  with 
a  ‘big  picture’  of  the  battle-space;  assisting  combat  aircraft  in  finding,  identifying,  and 
destroying  enemy  targets,  keeping  track  of  friendly  assets,  and  coordinating  air  refueling. 

Air  battle  management  functions  are  accomplished  at  the  operational  level  through  Air 
Operations  Centers  (AOC)  and  at  the  tactical  level  through  the  USAF  E-3  Airborne  Warning  and 
Control  System  (AWACS),  the  E-8  Joint  Surveillance  Target  Attack  Radar  (JSTARS),  the  USN 
E2C  Hawkeye  (see  Annistead  [2002]  and  Williams  [1997]  for  more  infonnation  on  airborne 
platforms),  and  a  variety  of  ground-based  Control  and  Reporting  Centers  (CRC).  At  the  tactical 
level,  functionality  is  contingent  upon  sensor  capabilities:  AWACS  and  JSTARS’  radar  and 
computer  subsystems  can  gather  and  present  expansive  and  detailed  battle-space  information  for 
air-to-air  and  air-to-surface  battle  management;  USN  E2C  Hawkeye  is  a  carrier-based  system 
with  similar  air-to-surface  and  air-to-air  sensor  capability  for  maritime  tactical  scenarios;  and 
CRCs  are  land  based,  short-range  systems  responsible  for  tactical  air  control  within  their  area  of 
responsibility  (AOR). 

Despite  the  variety  of  battle  management  platforms,  Weapons  Directors  (WDs;  i.e.,  Air 
Battle  Managers)  perform  analogous  tasks,  under  similar  conditions,  with  comparable  displays 
and  controls  (Knott  et  ah,  2007).  Thus  WDs  are  required  to  monitor  a  multiplicity  of 
simultaneous  communications  channels  under  conditions  of  moderate  to  high  ambient  cabin 
noise  while  performing  several  visual  and  manual  tasks  (Bolia  et  ah,  2005).  Additionally,  said 
tasks  are  performed  as  part  of  an  integrated  team  or  team  of  teams.  The  latter  requirement  is 
significant  because  it  emphasizes  the  importance  of  collaboration  for  the  achievement  of  tactical 
and  operational  goals  (Knott  et  ah,  2006). 

Currently,  WDs  collaborate  (i.e.,  monitor  and  transmit)  through  several,  often  overloaded 
voice  communication  channels,  within  a  field  of  moderate  to  high  ambient  platfonn  noise  (Bolia 
et  ah,  2005).  Thus  alternative  fonns  of  communication  such  as  visual  communication  could 
mitigate  the  above  issues  and  provide  a  meaningful  platform  for  collaboration.  Indeed,  language 
(i.e.,  spoken  and  written)  is  only  one  type  of  communication  among  other  types  of  sign  systems 
that  can  be  learned,  transmitted,  and  interpreted.  However,  it  is  unknown  as  to  how  the 
employment  of  non-language  based  communication  systems  utilized  in  conjunction  with 
language-based  systems  affects  team  performance.  Specifically,  for  the  air  battle  management 
domain,  detennining  whether  non-language  based  communication  modalities  reduce  the 
demands  of  a  saturated  voice  channel  can  contribute  to  the  development  of  more  effective 
collaborative  systems.  In  other  words,  an  understanding  of  the  relative  demands  (e.g., 
information)  of  various  communication  modalities  (e.g.,  visual  and  language-based)  can  reveal 
the  potential  advantages  and  disadvantages  of  utilizing  different  communication  modalities  in 
conjunction. 

Contemporary  semioticians  study  signs  (i.e.,  “...something  which  stands  to  somebody  for 
something  in  some  respect  or  capacity”;  Peirce,  1931)  as  part  of  meaningful  semiotic  ‘sign 
systems’.  According  to  semiotics,  meaningful  communication  can  be  transmitted  through  any 
medium  or  text  (i.e.,  a  message  or  assemblage  of  signs  that  has  been  recorded  in  some  way), 
including  verbal,  non-verbal,  or  both.  Additionally,  semioticians  maintain  that  the  multi-sensory 
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nature  of  human  experience  renders  every  representation  of  experience  subject  to  the  constraints 
of  media  involved.  Thus,  for  example,  the  flexible  medium  of  radio  communication  provides  a 
transient  signal  that  can  easily  be  missed,  misinterpreted,  or  forgotten.  Furthermore,  the 
communication  ‘signal’  transmitted  by  a  particular  medium  can  face  ‘interference’  by  various 
forms  of  extrinsic  ‘noise’.  Consequently,  for  example,  the  meaning  of  communication  over  radio 
channels  can  be  lost  through  a  field  of  background  ambient  noise. 

Communicating  meaningful  information  is  a  key  component  of  net-centricity.  Net-centric ity 
is  defined  as  the  “ability  to  get  the  right  information  to  the  right  people,  at  the  right  time,  using 
the  right  media,  in  the  right  language  and  at  the  right  level  of  detail”  (Satellite  Evolution  Global, 
2006).  Net-centricity  requires  appropriate  collaborative  systems  to  afford  users  the  ability  to 
discover,  access,  and  integrate  information  that  supports  mission  objectives.  In  a  net-centric 
environment,  WDs  will  have  more  information  at  their  fingertips  from  remote  sensors. 

However,  with  regard  to  WD  effectiveness,  the  relative  effects  of  increased  information 
availability  have  not  been  determined.  Thus,  it  is  unknown  as  to  whether  the  provision  of  more 
system-related  information  will  reduce  the  need  for  communication  (e.g.,  language-based  or 
visual);  increase/decreased  perceived  workload;  or  increase/decrease  mission-related 
performance. 

The  current  literature  on  collaboration  technologies  and  computer-mediated  communication 
focuses  almost  exclusively  on  language-based  or  verbal  communication  paradigms  within 
business  environments  with  the  goal  of  achieving  common  work  products,  such  as  decision 
consensus  (Knott  et  ah,  2007).  Thus,  the  applicability  of  these  studies  to  military  C2  domains, 
where  the  task  is  to  execute  a  mission,  is  limited.  The  purpose  of  the  present  investigation  is  to 
compare  the  performance  of  teams  using  radio  voice  communication  with  teams  utilizing  1)  a 
near  real-time  iconographic  digital  whiteboard  interface  and  2)  a  separable  digital  interface  that 
displays  low-level,  but  pertinent  information  (i.e.,  fuel  and  weapons  status)  as  a  supplementary 
means  of  communication  in  a  high-workload  military  C2  environment. 

Method 


The  DDD  Simulator 

The  Distributed  Dynamic  Decision-Making  (DDD;  see  www.Aptima.com)  software  is  a 
tool  for  creating  human-in-the-loop,  distributed,  multi-person  simulations.  The  DDD  was 
employed  to  create  a  set  of  Tactical  Air  Battle  Management  TABM  simulations  conveyed  to 
participants  through  a  tactical  display. 

Scenarios 

A  set  of  tactical  air  battle  management  (TABM)  scenarios  were  developed  for  this 
experiment  and  are  designed  to  require  a  team  of  two  Weapons  Directors  (WDs)  to  coordinate  to 
sustain  offensive  counter-air,  defensive  counter-air,  and  air  refueling  operations.  Weapons 
Directors  coordinate  operations  (i.e.,  intercept  threats  and  re-supply  assets  as  needed)  through 
communication,  sharing  of  fighter  assets,  and  allocating  two  tanker  assets.  The  scenarios  were 
presented  to  WDs  via  the  DDD  tactical  display  (see  Figure  1),  which  represented  the  area  of 
operations  with  friendly  assets  and  enemy  targets  shown  as  unique  symbols.  The  tactical  display 
afforded  a  real-time  representation  of  the  battle  space  from  which  WDs  were  able  to  monitor  and 
direct  simulated  air  operations. 
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The  tactical  display  divides  the  area  of  operations  into  three  regions  -  Gray,  Yellow  and 
Red  zones  -  that  represent  different  operational  areas.  The  Gray  zone  is  the  ‘kill  zone’,  the 
Yellow  zone  represents  friendly  airspace,  and  the  Red  zone  represents  the  friendly  region 
containing  ground  assets  (i.e.,  Ranger  units  and  an  air  base).  The  green  and  blue  symbols 
represent  friendly  fighter  assets  that  are  labeled  according  to  their  platfonn  type  (e.g.,  F-18)  and 
callsign  (e.g.,  Shaggy).  Additionally,  there  are  two  tanker  aircraft  (i.e.,  Air  Force  and  Navy)  for 
aerial  refueling  of  designated  fighter  assets.  Thus,  the  Air  Force  tanker  is  ‘equipped’  to  refuel  F- 
15s  and  F-16s  and  the  Navy  tanker  is  ‘equipped’  to  refuel  F-18s. 

The  scenario  is  a  10-minute  simulated  counter  air  operation  in  which  enemy  targets  enter 
the  green  zone  and  immediately  begin  moving  towards  friendly  territory.  Enemy  forces  have  the 
ability  to  attack  and  destroy  all  fighter  assets,  tankers,  the  air  base  and  the  infantry. 

Each  fighter  begins  the  scenario  with  weapons  resources  adequate  to  complete  two 
attacks  on  hostile  targets  (F15/F16/F18s  can  eliminate  MiGs;  only  FI 8s  can  eliminate  Su27s), 
and  with  a  randomly  assigned  quantity  of  fuel.  The  WD’s  task  is  to  choose  appropriate  asset- 
target  pairings  given  the  available  resources  for  each  asset,  communicate  the  asset-target  pairing 
decisions  to  friendly  fighter  assets,  and  prioritize  and  coordinate  weapons  re-supply  and  aerial 
refuelling  with  the  tankers. 


Figure  1.  The  WD’s  tactical  situation  display.  The  Green  and  Blue  WD  assets  are  colour- 
coded  and  labelled  with  a  call-sign.  The  blue  and  yellow  rings  represerit  a  fighter  asset’s 
sensor  range  and  vulnerability  to  attack,  respectively.  The  enemy  targets,  labelled  as 
MiG-2  5  s  or  Su-27s,  enter  the  simulation  from  the  right  of  the  display. 

Each  WD  is  assigned  a  separate  geographic  region  within  the  area  of  responsibility  and 
allocated  fighter  assets  to  manage.  The  area  of  responsibility  is  divided  into  two  prescribed 
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regions,  a  northern  (WD  Green’s  region)  and  a  southern  (WD  Blue’s  region),  and  their  division 
was  indicated  on  the  tactical  display  by  a  solid  black  horizontal  line.  Additionally,  asset  symbols 
were  color-coded  to  represent  asset  allocation  and  control  (e.g.,  green  assets  were  allocated  to 
WD  Green).  Although  each  WD’s  fighters  operate  primarily  within  his  or  her  region,  fighter 
assets  were  able  to  cross  regional  boundaries  if  necessary  to  provide  assistance  by  engaging  a 
target  for  which  the  other  WD’s  assets  had  insufficient  resources. 

The  WDs’  primary  responsibilities  include  relaying  tactical  information  to  their  assets, 
directing  assets  to  intercept  hostile  targets,  and  coordinating  aerial  re-supply  between  assets  and 
tankers.  To  do  this  effectively,  WDs  must  understand  the  capabilities  and  limitations  of  their 
operational  environment.  Within  the  simulation,  all  friendly  fighter  assets  had  the  same  fuel 
capacity.  However,  at  the  onset  of  each  mission  scenario,  each  asset  was  randomly  assigned 
different  fuel  loads.  Thus,  each  fighter  asset  had  to  be  refuelled  once  during  the  10-minute 
scenario.  All  assets  have  weaponry  adequate  for  two  attacks  before  they  must  be  re-supplied. 
Enemy  targets  were  differentiated  by  their  on-screen  representation  and  their  speed  of 
movement.  The  majority  of  enemy  targets  in  each  scenario  were  represented  by  a  red,  inverted 
“V.”  These  targets,  identified  as  “MiGs,”  were  slightly  slower  than  WD  fighter  assets  and  could 
be  pursued  and  intercepted  from  behind.  The  second  type  of  enemy  target,  identified  as  “Su- 
27s,”  was  represented  by  a  red  fighter  symbol.  Such  targets  were  slightly  faster  than  WD  fighter 
assets,  rendering  pursuit  ineffectual,  and  therefore  required  frontal  interception  by  fighter  assets. 
Each  time  a  MiG  was  intercepted  and  destroyed,  a  new  one  would  enter  the  airspace  to  replace  it 
from  the  right  side  of  the  display.  Thus,  the  number  of  targets  present  throughout  scenario  was 
deliberately  controlled. 

The  WDs  in  this  scenario  are  members  of  multiple  teams.  The  WDs  communicate 
directives  to  friendly  assets  through  Strike  Operators  (Green  Strike  and  Blue  Strike)  and  a 
Tanker  Operator.  The  two  Strike  Operators  play  the  role  of  multiple  fighter  pilots  and  manoeuvre 
assets  via  the  DDD  interface  as  directed  by  their  WD.  The  Tanker  Operator  manoeuvres  the  two 
tankers  (an  Air  Force  Tanker  and  a  Navy  Tanker)  to  refuel  and  resupply  assets  as  directed  by  the 
WDs.  In  this  experiment,  Strike  and  Tanker  Operators  were  highly  practiced  confederates  trained 
to  expertise  in  the  role  of  the  strike  fighter  and  tanker  operators.  As  such,  their  perfonnance  is 
related  to,  but  is  not  the  focus  of,  team  performance  in  this  experiment.  Instead,  the  primary 
focus  is  the  WDs’  task  performance. 

The  team  in  this  scenario  included  five  individuals:  Green  WD,  Blue  WD,  Green  Strike, 
Blue  Strike  and  the  Tanker  Operator.  The  two  WDs  have  all  decision  making  responsibility  and 
direct  and  manage  all  air  combat  operations,  coordinate  the  team,  and  act  on  infonnation  gleaned 
from  their  tactical  display  and  from  communication  with  the  other  operators.  The  Tanker 
Operator  and  Strike  Operators,  on  the  other  hand,  execute  directives  from  the  WDs  and  also, 
depending  on  experimental  condition,  provide  the  WDs  with  status  updates  on  their  assets,  such 
as  fuel  and  weapons  levels. 

The  WDs’  tactical  displays  provide  a  global  picture  of  the  simulated  battle  space, 
including  all  allied  and  enemy  entities.  The  Strike  Operators  are  able  to  see  all  friendly  air  or 
ground  assets,  but  see  enemy  aircraft  only  when  they  come  within  the  limited  range  of  their 
platform’s  sensors  (represented  by  a  blue  ring).  Thus,  they  have  limited  awareness  of  the  tactical 
picture  and  must  rely  on  the  WDs  to  vector  them  to  targets. 
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The  experiment  took  place  in  a  9.75  m  x  6.5  m  room  with  two  WDs  on  one  side  of  the 
room  and  the  Strike  and  Tanker  Operators  on  the  other  side,  facing  the  opposite  direction.  Each 
operator  had  a  17-inch  flat-panel  display  that  presented  the  tactical  display,  the  Virtual 
Whiteboard  tool  for  visual  communication,  and  the  Separable  Status  Display  (i.e.,  provides  fuel 
and  weapons  status  for  fighter  assets).  ModlOS®  Voice  Communicator  was  used  for  simulated 
network  radio  communication.  Each  of  the  WD’s  radios  comprised  one  communication 
frequency  for  speaking  with  their  Strike  Operator,  the  Tanker  Operator,  and  their  WD  partner. 
The  Virtual  Whiteboard  tool  provided  a  transparent  overlay  that  facilitated  visual  communication 
between  WDs  and  their  Strike  Operators  and  Tanker  Operator. 

The  Virtual  Whiteboard  is  a  domain-specific  graphical  collaboration  tool  tailored 
specifically  to  the  needs  of  prescribed  task  (e.g.,  Bolstad  &  Endsley  2005).  It  was  developed  to 
augment  linguistic  communication  in  the  air  battle  management  domain.  The  Virtual 
Whiteboard  affords  near  real-time  conveyance  of  tactical  display  and  iconographic  information 
(i.e.,  task-relevant  symbols  indicating  desired  actions  or  directives)  between  WDs. 

WDs  used  the  DDD  tactical  display  to  monitor  the  battle  and  then  used  the 
communications  software  to  issue  directives  to  the  Strike  and  Tanker  Operators.  The  Strike 
Operators  and  Tanker  Operators  used  the  DDD  interface  to  operate  the  strike  assets  and  tankers 
and  to  retrieve  information  about  their  assets.  Radios  were  operated  with  a  footswitch  for  all 
operators.  Participants  wore  headsets  throughout  the  experiment  and  white-noise  was  generated 
in  the  lab  at  approximately  75  dBA  during  all  trials.  The  purpose  of  the  white-noise  was  to 
simulate  the  noise  of  an  AWACS  or  JSTARS  platform,  and  to  prevent  participants  from 
communicating  with  each  other  except  by  the  means  provided. 

Secondary  task.  The  Coordinate  Response  Measure  (CRM;  Bolia  et  ah,  2000)  consists  of 
2,048  unique  phrases  of  the  form  “Ready  (CALL  SIGN),  go  to  (COLOR)  (NUMBER)  now,”  for 
example  “Ready  Baron,  go  to  White  Four  now”.  Eight  unique  voices  -  four  male  and  four 
female-  read  each  of  256  possible  Call  Sign,  Color,  Number  combinations,  and  each  file  is 
approximately  1.5  seconds  in  duration.  Use  of  the  Color/Number  pair  of  the  CRM  stimuli 
provides  a  secondary  task  requiring  participants  to  attend  not  only  to  identifying  the  voice  that  is 
speaking  but  also  to  the  informational  content  of  the  message  they  are  hearing  -  a  quality  similar 
to  real-world  interactions. 

Procedure 

Prior  to  the  experiment,  all  participants  (WDs)  completed  a  one  day,  four  hour  training 
session  in  which  they  were  trained  on  the  scenario,  the  radio  software,  Virtual  Whiteboard  tool, 
Separable  Status  Display,  and  then  completed  eleven  10-minute  practice  trials  of  the  DDD 
scenario.  The  trainer  informed  participants  that  the  purpose  of  the  study  was  to  evaluate  how 
teams  used  communication  technology  to  work  together  and  that  they  would  be  playing  a 
computer  game  that  required  teamwork  to  meet  the  game’s  objectives.  Additionally,  WDs  were 
trained  on  and  practiced  communication  brevity  for  voice  communications.  Brevity  training  was 
critical  to  minimize  irrelevant,  unnecessarily  lengthy  and/or  confusing  verbal  statements. 

Participants  were  also  trained  on  the  specific  objectives  and  rules  of  the  mission,  and 
were  instructed  that  the  performance  of  the  team  would  be  measured  for  each  trial  based  on  how 
well  they  met  their  objectives  and  followed  the  rules.  Mission  objectives  included:  destroy  as 
many  hostile  aircraft  as  quickly  as  possible;  do  not  allow  hostile  aircraft  to  enter  friendly 
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territory;  protect  the  Air  base  and  the  infantry  from  enemy  attack;  protect  the  Air  Force  and  navy 
Tankers  from  enemy  attack;  and  keep  as  many  fighters  airborne  for  as  long  as  possible.  Mission 
rules  included:  refuel  aircraft  at  appropriate  Tanker  (i.e.,  F18s  at  Navy  Tanker;  F15s  and  F16s  at 
Air  Force  Tanker);  refuel  at  Air  Base  only  if  airborne  refuelling  is  not  possible;  F15/F16/F18s 
can  attack  MiGs,  whereas  only  FI 8s  can  attack  Su27s;  and  the  Green  WD  is  responsible  for 
defending  the  Northern  region,  whereas  Blue  WD  is  responsible  for  defending  the  Southern 
region.  Upon  completing  training,  WDs  were  administered  a  training  quiz.  Subjects  were 
required  to  obtain  a  score  of  100%  on  the  training  quiz  before  moving  on  to  experimental 
sessions  (subjects  were  permitted  to  re-take  the  quiz  if  minimal  score  was  not  obtained). 

Upon  completing  training,  WDs  returned  the  next  day  for  the  experimental  session.  In  this 
session  they  completed  sixteen  experimental  trials.  Each  experimental  trial  comprised  of 
conducting  the  TABM  scenario  for  a  10-minute  run.  For  each  experimental  condition,  subjects 
were  exposed  to  two  practice  trials  (without  the  CRM  secondary  task)  and  two  non-practice  trials 
(with  the  CRM  secondary  task).  During  non-practice  trials,  subjects  were  required  to  respond 
(on  a  8x4  grid;  see  above)  to  CRM  auditory  stimuli  that  indicated  “Ready  baron,  (COLOR) 
(NUMBER)”.  After  each  trial,  participants  completed  several  subjective  instruments  designed  to 
assess  mental  workload  and  satisfaction  with  collaborative  technologies.  Participants  were  given 
one  20-minute  rest  period  after  they  had  completed  half  of  the  trials.  All  major  simulation  events 
(e.g.,  the  occurrence  and  outcome  of  attacks,  refuelling  events,  etc.)  were  recorded  in  data  logs 
for  later  analysis.  In  addition,  video  of  the  all  team  members  and  all  voice  and  chat 
communications  were  recorded. 

Experimental  Design 

There  were  two  levels  of  Communications  Modality  (voice-only  and  voice  and  Virtual 
Whiteboard  tool),  and  two  levels  of  status  display  (Separable  Status  Display  and  no-Separable 
Status  Display).  These  independent  variables  were  combined  factorially  yielding  a  2  x  2  within- 
subjects  design.  Each  team  completed  two  10-minute  trials  under  the  four  experimental 
conditions.  Prior  to  conducting  the  each  block  of  two  trials,  teams  complete  two  10-minute 
practice  trials  for  the  relevant  condition.  The  order  of  conditions  was  counterbalanced  across 
trials.  In  the  voice  and  voice-and- Virtual  Whiteboard  condition,  all  members  of  the  team  were 
given  the  option  to  communicate  using  voice,  Virtual  Whiteboard,  or  a  combination  of  the  two. 

Subjective  Measures.  A  modified  version  of  the  NAS  AS  TLX  (Hart  &  Staveland,  1987) 
was  used  to  evaluate  perceived  task  demands.  In  the  modified  version,  participants  were  asked  to 
estimate  the  workload  for  the  team  rather  than  rating  their  individual  workload. 

Team  Performance.  Measures  of  team  performance  on  the  TABM  scenario  trials  were  1) 
the  percentage  of  enemy  targets  that  were  allowed  to  penetrate  friendly  airspace,  2)  the 
percentage  of  high  value  assets  destroyed  (i.e.,  the  air  base,  infantry  units,  and  tanker  aircraft),  3) 
the  percentage  of  fighter  assets  that  were  destroyed,  4)  average  time  to  prosecute  a  target. 

Results 


Team  Performance 

As  mentioned  above,  several  measures  of  team  perfonnance  were  assessed  for  each  trial. 
There  were  no  significant  differences  on  the  primary  measures  for  team  performance.  However, 
analyses  indicated  a  significant  trend  for  communication  medium  TfiT,  7)  =  5.45,  p  =.052  (see 
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Figure  2)  on  the  average  time  average  time  to  prosecute  a  target.  This  measure  was  the  average 
elapsed  time  (in  seconds)  from  when  an  enemy  target  was  presented  on  the  tactical  display  until 
it  was  destroyed.  Analyses  revealed  that  average  time  of  enemy  prosecution  was  faster  during  the 
Virtual  Whiteboard  conditions  compared  to  voice-only  conditions  (see  Figure  2). 
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Figure  2.  Mean  time  of  enemy  prosecution  as  a  function  of  communication  modality. 
Error  bars  represent  one  standard  error  of  the  mean  in  each  direction. 


Workload 

As  mentioned  above,  subjective  workload  (i.e.,  NASA-TLX)  was  assessed  for  each  trial. 
Unweighted  NASA-TLX  ratings  were  submitted  to  a  5  (Subscale)  x  2  (Communication 
Modality)  x  2  (SSD)  repeated  measures  analysis  of  variance.  The  analysis  indicated  a  main 
effect  for  communication  medium  F(  1 ,  7)  =  36.92,  p  <  .001  (see  Figure  3).  Also,  a  main  effect 
for  SSD  presence  was  obtained,  F{  1,  7)  =  10.10,  p  <  .05,  as  (see  Figure  4).  Analyses  revealed 
that  mean  subjective  workload  ratings  were  lowest  for  Virtual  Whiteboard  conditions  and 
conditions  without  the  SSD  whereas  mean  subjective  workload  ratings  were  highest  for  voice- 
only  conditions  and  conditions  with  the  SSD. 
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Figure  3.  Mean  unweighted  NASA-TLX  ratings  as  a  function  of  communication  modality. 
Error  bars  represent  one  standard  error  of  the  mean  in  each  direction. 
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Figure  4.  Mean  unweighted  NASA-TLX  ratings  as  a  function  of  SSD  presence.  Error- 
bars  represent  one  standard  error  of  the  mean  in  each  direction. 

Voice  Communication 

Communication  frequency  and  duration  were  analysed  to  determine  1)  if  the  utilization  of 
a  visual  communication  modality  (i.e,.  Virtual  Whiteboard)  reduced  the  demands  of  a  voice 
channel,  and  2)  whether  increased  information  availability  (i.e.,  SSD)  reduced  the  need  for 
communication  (e.g.,  language-based  or  visual). 

Communication  frequency  was  the  number  of  radio  transmissions  between  team 
members  during  the  experimental  trials.  Analyses  of  communication  frequency  indicated  a  main 
effect  for  communication  medium,  F{  1,  7)  =  68.09,  p  <  .0001,  and  presence  of  SSD,  F{  1,  7)  = 
9.86,  p  <  .05  (see  Figures  5  and  6).  Analyses  reveal  that  frequency  of  voice  communication  was 
reduced  when  teams  could  use  the  Virtual  Whiteboard  medium  and  when  the  SSD  was  present. 
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Figure  5.  The  mean  number  of  radio  transmissions  per  experimental  trial  as  a  function  of 
communication  modality.  Error  bars  represent  one  standard  error  of  the  mean  in  each 
direction. 
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Figure  6.  The  mean  number  of  radio  transmissions  per  experimental  trial  as  a  function  of 
communication  modality.  Error  bars  represent  one  standard  error  of  the  mean  in  each 
direction. 


Communication  duration  was  calculated  as  the  total  duration  (in  seconds)  of  voice 
communication  over  the  radio  for  each  10-minute  experimental  trial.  Analyses  of 
communication  duration  indicated  a  main  effect  for  communication  medium,  F(  1,7)  =  79.07,  p 
<  .0001  (see  Figure  7).  Analyses  reveal  that  duration  of  voice  communication  was  reduced  when 
teams  could  use  the  Virtual  Whiteboard  medium. 
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Figure  7.  The  mean  duration  of  voice  communication  over  the  radio  as  a  function  of 
communication  modality.  Error  bars  represent  one  standard  error  of  the  mean  in  each 
direction. 

Secondary  Task 

The  secondary  task  (i.e.,  CRM)  employed  is  indicative  of  the  ‘chatter’  present  over  radio 
communication  channels  in  the  air  battle  management  domain.  Analyses  of  CRM  performance 
indicate  a  significant  interaction  between  communication  medium  and  SSD  presence  F(\ ,  14.8) 
=  4.80,  p  <  .05.  Analyses  reveal  that  the  best  CRM  performance  was  achieved  during  Virtual 
Whiteboard  conditions  without  the  presence  of  the  SSD  whereas  the  worst  CRM  performance 
was  achieved  during  Virtual  Whiteboard  conditions  with  the  presence  of  the  SSD  (see  Figure  8). 
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Figure  8.  The  Percentage  of  correct  CRM  responses  as  a  function  of  communication 
medium  and  presence  of  the  SSD.  Error  bars  represent  one  standard  error  of  the  mean 
in  each  direction. 

Discussion 

The  means  through  which  communication  is  supported  is  an  important  part  of  dynamic 
decision-making  and  managing  team  interdependence  (Rasmussen,  Brehmer,  &  Leplat,  1991). 
Indeed,  the  demands  of  coordination  within  a  complex  C2  domain  place  constraints  on  the  type 
of  communication  media  utilized.  Thus,  within  the  AWACS  BMC2  domain,  teams  must  work 
through  a  dynamic  and  complex  electronic  battlefield  wherein  “. .  .the  time  from  detection  to 
commitment  may  be  measured  in  fractions  of  seconds”  (Rochlin,  1996;  p.  205).  Therefore, 
within  the  AWACS  BMC2  domain,  effective  collaborative  systems  must  support  rapid 
dissemination  of  relevant  information.  To  be  sure,  AWACS  BMC2  teams  will  adapt  to  the 
functional  constraints  of  the  work  domain  if  those  constraints  are  visible.  Consequently,  an 
effective  communication  medium  must  afford  agents  sufficient  control  of  the  complex  AWACS 
domain  by  behaving  like  a  window  through  which  agents  can  share  relevant  knowledge  (Flach  & 
Bennett,  1996;  Schwartz,  Flach,  Nelson,  &  Stokes,  2007). 

According  to  semiotics,  both  visual  and  verbal  communication  can  afford  agents 
sufficient  control  of  complex  domains.  Indeed,  the  dimensionality  (e.g.,  visual  vs.  verbal)  of 
communication  that  is  most  effective  in  revealing  domain  constraints  may  be  informed  by  the 
specific  tasks  involved  (Wainfan  &  Davis,  2004).  Thus,  for  a  complex  domain,  such  as  AWACS 
BMC2,  wherein  flexible  communication  is  necessary  in  order  to  cope  with  dynamic  situations,  a 
multi-modal  collaborative  system  can  support  teams  and  help  ensure  mission  success. 
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Current  literature  on  collaboration  technologies  focuses  almost  exclusively  on  mono- 
modal,  language-based  or  verbal  communication  paradigms  within  business  domains.  However, 
the  effectiveness  of  non-language  based  collaborative  systems  within  C2  domains  has  been 
neglected.  The  purpose  of  this  research  was  to  detennine  the  effectiveness  of  a  non-language 
based  visual  collaborative  tool  (i.e.,  Virtual  Whiteboard)  within  the  AWACS  BMC2  domain. 
Additionally,  the  effectiveness  of  a  non-verbal  status  display  system  (i.e.,  Separable  Status 
Display;  SSD)  was  evaluated  to  determine  how  increased  infonnation  availability  impacted 
performance  -  whether  the  provision  of  more  system-related  information  reduced  the  need  for 
communication  (e.g.,  language-based  or  visual);  increase/decreased  perceived  workload;  or 
increase/decrease  mission-related  performance. 

Overall,  results  reveal  that  the  Virtual  Whiteboard  tool  improved  the  effectiveness  of 
AWACS  BMC2  teams.  First,  results  indicated  that  using  only  voice  (i.e.,  linguistic) 
communication  resulted  in  higher  subjective  ratings  of  workload  compared  to  voice 
communication  augmented  with  visual  communication  (i.e.,  Virtual  Whiteboard).  The 
implications  of  this  can  be  seen  in  the  CRM  results  wherein  performance  was  better  using  the 
Virtual  Whiteboard  tool.  Presumably,  subjects  reported  lower  workload  ratings  during  trials 
with  the  Virtual  Whiteboard  tool  available  because  visual  communication  offset  the  linguistic 
communication  demands  (e.g.,  saturation)  evidenced  by  the  communication  analysis. 

Specifically,  when  the  Virtual  Whiteboard  tool  was  utilized,  frequency  and  duration  of 
communication  was  reduced.  Second,  results  indicate  that  the  Virtual  Whiteboard  tool  reduced 
the  amount  of  time  each  team  spent  prosecuting  targets.  Within  the  time-critical  domain  of  air 
battle  management,  shedding  seconds  in  the  identification  and  prosecution  of  critical  targets  is  a 
meaningful  performance  enhancement.  To  be  sure,  in  a  battle  situation,  the  seconds  exhausted 
prosecuting  individual  targets  are  compounded  by  the  emergence  of  new  potential  targets.  Thus, 
if  the  number  of  targets  entering  the  area  of  responsibility  (i.e.,  the  screen)  was  not  controlled 
(i.e.,  in  this  study,  number  of  targets  was  controlled)  there  would  have  be  a  ‘build-up’  of  targets 
during  voice-only  trials  (i.e.,  in  real-world  scenarios).  Furthermore,  in  real-world  battle 
management  domains,  emerging  situations  can  change  dynamically  within  seconds.  Thus,  the 
ability  of  the  WD  to  respond  effectively  within  a  very  small  window  of  time  is  critical. 

Results  were  mixed  for  the  SSD.  On  the  negative  side,  presence  of  the  SSD  increased 
teams’  subjective  workload  ratings.  However,  presence  of  the  SSD  decreased  the  frequency  and 
duration  of  communication.  It  can  be  argued  that  the  latter  effect  occurred  because  the 
information  made  available  by  the  SSD  reduced  the  amount  of  information  required  over 
communication  modalities  (i.e.,  fuel  and  weapons  status).  However,  as  mentioned  above  in  terms 
of  net-centricity,  information  should  be  available  to  the  right  people,  at  the  right  time,  in  the  right 
form.  Perhaps,  the  increased  perceived  workload  can  be  attributed  to  the  latter  aspect  of  net- 
centricity;  that  of  form. 

The  status  display  employed  in  this  study  was  a  separable  status  display.  According  to 
Flach  &  Bennett  (1992;  also  Pomerantz,  1986),  there  are  three  different  types  of  displays  - 
separable  (separate  elements  for  separate  data  variables),  integral  (high  degree  of  interaction 
among  elements;  e.g.,  color  is  composite  of  hue  and  brightness),  and  configural  (i.e.,  elements 
maintain  separate  status  and  interact  to  produce  emergent  features).  Research  conducted  on 
separable  and  configural  displays  concludes  that  configural  displays  are  better  for  integration 
tasks  (where  an  agent  must  consider  multiple  elements/variables  at  once)  and  focused  tasks 
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(considering  each  element/variable  separately;  Flach  &  Bennett,  1992).  The  key  to  configural 
displays  is  that  they  adequately  specify  the  process  domain  that  must  be  controlled  -  in  other 
words,  the  semantics  (or  meaning)  of  the  domain  should  be  apparent.  Thus,  perhaps,  the  form  of 
the  infonnation  presented  in  the  separable  status  display  utilized  in  this  study  did  not  clearly 
specify  the  dimensions  of  system  that  needed  to  be  controlled.  Perceived  workload  ratings 
indicate  that  subjects  experienced  higher  workload  when  they  were  required  to  make  sense  out  of 
information  presented  in  a  separable  display  (as  opposed  to  hearing  relevant  information  through 
voice  communication).  Future  research  should  consider  the  impact  of  a  configural  status  display, 
wherein  symmetrical  geometric  forms  represent  meaningful  information. 
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